Skip to content

Phase 3: staging zarr conversion pipeline - #1200

Merged
neuromechanist merged 16 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1180-phase3-staging-pipeline
Sep 2, 2026
Merged

neuromechanist merged 16 commits into
feature/issue-1181-epic-zarr-servingfrom
feature/issue-1180-phase3-staging-pipeline

Conversation

@neuromechanist

@neuromechanist neuromechanist commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add --test and --print-config to scripts/zarr/hallu-zarr.sh, giving the Zarr converter a second, independently-stated instance that converts real BIDS recordings against nemar-db-dev / nemar-dev / zarr-test.nemar.org instead of prod. --test is a pre-pass over "$@" that exports test defaults (API_BASE, TEST_API_URL, S3_BUCKET, ZARR_AWS_PROFILE, ZARR_STATE_DIR, ZARR_WORK_DIR, ZARR_DRIVER_REF, ZARR_JOBS=4) only for variables still unset, before the existing Config block resolves, and records TEST_PREPASS_SEEN as the raw, can't-be-swallowed record of "--test was in argv".
  • Guard rails (six conditions): --test refuses to run if S3_BUCKET=nemar, API_BASE/TEST_API_URL name the prod host api.nemar.org (case-insensitive, scheme/port/path/trailing-slash agnostic — api-test.nemar.org is still allowed), AWS_PROFILE=nemar-zarr, or STATE_DIR/WORK_DIR normalize (trailing/doubled slashes collapsed, realpath-resolved when the dir exists) to the prod defaults — a stale prod value exported alongside --test stops the run instead of silently pointing at prod. The guard block keys off TEST_PREPASS_SEEN, not the arg parser's TEST_MODE, so it cannot be suppressed by a parsing quirk in some other flag.
  • --dataset/--limit now refuse a value starting with -- (same guard --requeue already had), closing a hole where --dataset --test/--limit --test swallowed the --test token as the flag's own value.
  • A plain (non---test) run explicitly unsets an ambient TEST_API_URL left over from an earlier --test session, logging one line when it does, so a real prod conversion can never depend on stale shell state.
  • --print-config prints every resolved config value (secrets only as present/absent), including the operation flags (ONLY_DATASET, LIMIT, REQUEUE, BACKFILL_DIR_FORMATS, PREVIEW_ENGINE_BUMP, EXECUTE), and exits 0 before touching the filesystem, network, or lock, in every documented flag order/combination.
  • Deviation from the brief, both authorized by design decision 5 ("thread hardcoded prod-specifics through existing plumbing") and found live during verification: convert_dataset()'s metadata-clone step shells out to the separate nemar CLI binary, whose API-base resolution (src/lib/api/client.ts getApiUrl()) is independent of $API_BASE — it reads TEST_API_URL first, then a stored account config, then defaults to api.nemar.org. This was missed by issue Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180's own env-var inventory (scoped to this script and the Python driver/queue) and surfaced as Dataset not found on the first live --test --dataset attempt, because the CLI was looking the dataset up in prod. Fixed by having --test also export TEST_API_URL — the same existing hook AGENTS.md's staging-auth section already documents, not a new mechanism.
  • pytest suite scripts/zarr/test_hallu_zarr_config.py (29 tests, real subprocess execution of the script, no mocks — up from 10 after a three-reviewer pass reproduced and closed four guard-bypasses live), wired into CI's zarr-python-test job.
  • Docs: .context/systems-inventory.md §3.3 corrected to match the live crontab ground truth (prod cron invokes the nemar-cli clone's own copy of hallu-zarr.sh, not a hand-placed standalone file; hand-placement is bootstrap-only) and §3.4 (paths, credentials table, bootstrap/run-by-hand recipes, cron line, what is/isn't shared with prod, live verification results); .context/validated_workflows.md §9 (bootstrap-and-run recipe with the exact commands that worked and the gotchas hit, including a note that later runs must invoke the clone's copy, not the bootstrap copy); ADR 0029 gets a dated amendment note (decision text left as originally written) pointing at the corrected description; one-sentence pointer added to AGENTS.md's Zarr paragraph.
  • No changes were needed in zarr_queue.py/generate_zarr.py themselves — reconcile --api-base and generate_zarr.py --bucket already thread everything through as issue Zarr conversion pipeline for the staging data plane (zarr-test.nemar.org) #1180 said. The exemplar-fleet copy path is untouched — it's the copy phase of nemar admin exemplar create --include-derived, not a standalone copy subcommand.

Live verification (2026-09-02)

Ran on Hallu, bootstrapped from this branch (ZARR_DRIVER_REF=feature/issue-1180-phase3-staging-pipeline), against dataset xx099905 ([TEST COPY] BNCI 2014-004 Motor Imagery dataset, an exemplar-fleet dataset public in dev, never converted there before):

hallu-zarr.sh --test --dataset xx099905
  • Metadata cloned from api-test.nemar.org, all 5 raw .bdf recordings streamed from s3://nemar-dev/xx099905/objects/, converted, and pushed to s3://nemar-dev/xx099905/zarr/.
  • https://zarr-test.nemar.org/xx099905/zarr/index.json served the fresh result immediately: store_count: 5, updated_utc: 2026-09-02T16:08:31Z, source_commit matching the clone's HEAD.
  • Isolation confirmed: prod zarr-queue.db mtime and .nm-zarr.lock unchanged before/after; aws s3api list-objects-v2 --bucket nemar --prefix xx099905/zarr/ (my own admin credentials) returned zero objects; api.nemar.org/datasets/xx099905 404s (the id exists only in dev).
  • Webhook callback skipped exactly as designed: no .zarr-secrets.env exists yet at /mnt/local/zarr-state-test/, the script logged the non-fatal warning, and api-test.nemar.org/datasets/xx099905's zarr_status/zarr_store_count/zarr_converted_at stayed null — S3/index.json (what the viewer actually reads) advanced regardless.
  • Never ran the prod script, never touched /mnt/local/zarr-state/, never --execute'd any requeue against prod.

Review round 2 (2026-09-02)

Three reviewers ran the script directly and reproduced four live guard bypasses; all four are fixed, with regression tests, plus the doc/wording follow-ups they flagged:

  1. --dataset --test / --limit --test swallowed --test as the flag's value, leaving TEST_MODE unset and the guard block silent even though the pre-pass had already applied test defaults over an ambient prod value. Fixed both ways: the value parser now refuses a ---prefixed value, and the guard block was moved off TEST_MODE onto the pre-pass's own TEST_PREPASS_SEEN.
  2. STATE_DIR/WORK_DIR guards were exact-string comparisons; a trailing or doubled slash slipped past. Now normalized (slash-collapsed, realpath-resolved when the directory exists) on both sides, and WORK_DIR is guarded at all for the first time.
  3. API_BASE/TEST_API_URL guards were case-sensitive substring matches; https://API.NEMAR.ORG slipped past. Now a case-insensitive, host-only comparison (scheme/port/path/trailing-slash stripped) that still allows api-test.nemar.org.
  4. A TEST_API_URL left exported from an earlier --test session would leak into a later plain prod run's nemar CLI call. A plain run now explicitly unsets it, logging once when it does.
  5. Doc/wording fixes: CI comment guard-rail count, §3.3 cron table and "Deploying a converter change" corrected to the live crontab ground truth (prod cron invokes the clone's own copy of hallu-zarr.sh, hand-placement is bootstrap-only), ADR 0029 dated amendment, "copy --include-derived" reworded to "the copy phase of nemar admin exemplar create --include-derived" (no standalone copy subcommand exists), the .nm-zarr.log table row reworded to note only the wrapper's own lines carry the [test] prefix, the AGENTS.md pointer reworded ("converts into nemar-dev's S3 zarr prefix, served at zarr-test.nemar.org"), the IAM credentials row dated, and a new gotcha in validated_workflows.md §9 about invoking the clone's copy (not the bootstrap copy) on later runs.

All of it re-verified live against the real --test guard rails on the actual script (not just the pytest suite) before pushing.

What remains for ops

  • Install the nightly cron line (.context/systems-inventory.md §3.4): 15 3 * * * /mnt/local/zarr-state-test/nemar-cli/scripts/zarr/hallu-zarr.sh --test >> /mnt/local/zarr-state-test/.nm-zarr-cron.log 2>&1.
  • Place .zarr-secrets.env (dev NEMAR_WEBHOOK_TOKEN) at /mnt/local/zarr-state-test/ so the zarr-ready callback stops being skipped and D1 zarr_status starts advancing on the test instance.
  • Follow-up for nemarOrg/website (not edited here): the website's AGENTS.md staging section should now note that zarr-test.nemar.org can carry real re-conversions, not just server-side-copied bytes from the exemplar fleet.

Test plan

  • bash -n scripts/zarr/hallu-zarr.sh
  • shellcheck --severity=warning scripts/zarr/hallu-zarr.sh
  • cd scripts/zarr && uv run --with pytest pytest test_hallu_zarr_config.py -v (29 passed)
  • python3 scripts/zarr/test_zarr_queue.py, test_generate_zarr.py, test_purge_non_raw_stores.py, test_patch_power_line_frequency.py all still green (no regressions)
  • Live conversion against nemar-dev/api-test.nemar.org/zarr-test.nemar.org (see above)
  • Prod isolation confirmed (queue db, lock, prod bucket, prod catalog)
  • All four review-round-2 guard bypasses re-reproduced against the real script and confirmed fixed before pushing (not just via pytest)

Closes #1180
Part of epic #1181

@github-actions

github-actions Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

📦 Package Preview

Your package is ready to test!

bun install -g nemar-cli@PR1200

Version: 0.9.12-PR1200.743b742
Commit: 743b742


🤖 Updated on each push.

@neuromechanist
neuromechanist merged commit 4057cc4 into feature/issue-1181-epic-zarr-serving Sep 2, 2026
12 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-1180-phase3-staging-pipeline branch September 2, 2026 17:04

This branch was previously deployed

1 inactive deployment
npm — 743b742a Deployed Sep 2, 2026 by neuromechanist via publish #2170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant